{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TorchSparse for OpenPCDet Plugin demo\n",
    "You can run the cells below to run the evaluation of TorchSparse integrated MMDetection3D models. \n",
    "\n",
    "## Dependencies\n",
    "- Conda\n",
    "- OpenPCDet installation: Please follow the [OpenPCDet documentation](https://github.com/open-mmlab/OpenPCDet/blob/master/docs/INSTALL.md). \n",
    "- Pre-process the datasets required by OpenPCDet ([see here](https://github.com/open-mmlab/OpenPCDet/blob/master/docs/GETTING_STARTED.md)). \n",
    "- TorchSparse installation.\n",
    "- Install TorchSparse plugin for OpenPCDet\n",
    "    1. Clone this repository\n",
    "    2. Define the environment variable `PCDET_BASE` to point to the installation path of OpenPCDet.\n",
    "    3. Go to `examples/openpcdet` and run `pip install -v -e .`\n",
    "\n",
    "## Note\n",
    "1. In some cases, you need to turn off PyTorch JIT compile to avoid JIT errors when using the OpenPCDet framework. [This post](https://discuss.pytorch.org/t/turning-pytorch-jit-mode-on-off-dynamically/31288/4) tells you how to turn it off. Typically, you just need to go to the `.py` file reporting this error and add the following line in the import section:\n",
    "```python\n",
    "import torch\n",
    "torch.jit._state.disable()\n",
    "```\n",
    "2. There is no way to replace the layers of a OpenPCDet model with the `cfg_options` like in the `mmdetection3d`, and it also doesn't have the option to register a customized layer. So, for the demonstration purposes, the torchsparse modules are monkey-patched to add the corresponding TorchSparse layers when the plugin is imported, and they are selected with the modified conguration file (under `examples/openpcdet/cfgs`) provided in the example. \n",
    "3. Modify the dataset path in the model config: The dataset config file path in the OpenPCDet's model config file is a relative path that is valid only if the you are running the evaluation under the `tools` directory. \n",
    "```yaml\n",
    "DATA_CONFIG: \n",
    "    _BASE_CONFIG_: cfgs/dataset_configs/kitti_dataset.yaml\n",
    "```\n",
    "It won't work if the script is being executed somewhere else, and it will raise the error `FileNotFoundError: [Errno 2] No such file or directory: 'cfgs/dataset_configs/kitti_dataset.yaml'`. Thus it needed to be changed into an absolute path to the dataset config in the model config and plug-in model config. The `DATA_PATH` pointing to kitti_dataset.yaml in the TorchSparse integrated model configs is automatically inferred and updated in `cfgs` during the plugin installation (it asked to provide the path `PCDET_BASE` as an environment variable). \n",
    "4. Then, you need to change the data root in the OpenPCDet's dataset config file to be the full path of the corresponding dataset root, for the same reason. For example, for Kitti models, you need to change `DATA_PATH: '../data/kitti'` in `OpenPCDet/tools/cfgs/dataset_configs/kitti_dataset.yaml` to point to the absolute path of the dataset base. \n",
    "\n",
    "## Steps\n",
    "1. Install the dependencies. \n",
    "2. Specify the base pathes and model registry.\n",
    "3. Activate the plugin: go to `OpenPCDet/tools/test.py` and add `import pcdet_plugin` as the last import statement. \n",
    "4. Run demo. \n",
    "5. Print the evaluation results. \n",
    "\n",
    "## Available models\n",
    "- Kitti\n",
    "    - SEOND\n",
    "    - PV_RCNN\n",
    "    - Part-A2\n",
    "- NuScenes\n",
    "    - VoxelNeXt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib.util\n",
    "import sys, os\n",
    "from pathlib import Path\n",
    "import subprocess\n",
    "\n",
    "# Loading the model converter to this notebook. \n",
    "# Define the relative path to the file\n",
    "relative_path = \"../converter.py\"\n",
    "file_path = Path().resolve() / relative_path\n",
    "\n",
    "# Add the directory containing the file to sys.path\n",
    "sys.path.append(str(file_path.parent))\n",
    "\n",
    "# Load the module\n",
    "spec = importlib.util.spec_from_file_location(\"convert_weights\", str(file_path))\n",
    "converter = importlib.util.module_from_spec(spec)\n",
    "spec.loader.exec_module(converter)\n",
    "\n",
    "converter = getattr(converter, \"convert_weights\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check that the model converter is successfully loaded\n",
    "print(converter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify the Paths\n",
    "To run this demo, you need to provide the following paths:\n",
    "1. `openpc_path`: OpenPCDet installation path.\n",
    "2. `openpc_model_base_path`: Input pretrained weight path. \n",
    "3. `torchsparse_model_base_path`: Output pretrained weight path.\n",
    "4. `openpc_cfg_base_path`: OpenPCDet configuration files base path.\n",
    "    - This configuration file is required in the model conversion. Specifically, it use the original configuration file to create a model to identify the Sparse Conv modules, and convert the weights for only those modules. If you installed the OpenPCDet on source, then it should be `tools/cfgs` folder in its repo. \n",
    "5. `torchsparse_cfg_base_path`: The path of the configuration file of TorchSparse integrated models. This configuration is the same as the config file in the OpenPCDet repository, except certain model layers is replaced with TorchSparse integrated modules. Default to be the `cfgs` folder in this example folder.  \n",
    "6. Conda environment name: this demo initialize a sub-shell to execute the demo with `subprocess`. So you need to specify the name of the conda environment that you want to use to run the demo. \n",
    "\n",
    "For paths 2, 3, 4, and 5, we expect you to organize them by having a base path and put the checkpoint/configurations files of different models under the same basepath. For example, for the input pertrained weight path, the file structure looks like: \n",
    "\n",
    "```text\n",
    "/home/ubuntu/openpc_model_base/                      \n",
    "├── SECOND/                 \n",
    "│   └── SECOND_Checkpoint.pth\n",
    "├── PV-RCNN/\n",
    "│   └── PV-RCNN_Checkpoint.pth\n",
    "└── VoxelNeXt/\n",
    "    └── VoxelNeXt_Checkpoint.pth\n",
    "```\n",
    "In this case, you need to configure the `openpc_model_base_path` to be `/home/ubuntu/openpc_model_base` and in the SECOND's registry entry, set `ckpt_before` to be `SECOND/SECOND_Checkpoint.pth`. \n",
    "\n",
    "In addition to the paths, we also need you to specify:\n",
    "1. SpConv version of the original model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_name = \"torchsparse\"\n",
    "\n",
    "base_paths = {\n",
    "    'openpc_path': None,\n",
    "    'openpc_model_base_path': None,\n",
    "    'torchsparse_model_base_path': os.path.joinin(os.path.abspath(''), \"converted_models\"),\n",
    "    'openpc_cfg_base_path': None,\n",
    "    'torchsparse_cfg_base_path': os.path.joinin(os.path.abspath(''), \"cfgs\")\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function to run a single demo is defined below. Based on the configuration dictionary you provid, it convert the model weights then use the `tools/test.py` in the `OpenPCDet` repo to run the model evaluation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def openpc_single_demo(registry_entry, base_paths, convert=True):\n",
    "\n",
    "    assert os.path.isdir(base_paths['openpc_model_base_path']), f\"OpenPCDet model base path {base_paths['openpc_model_base_path']} does not exist.\"\n",
    "    assert os.path.isdir(base_paths['torchsparse_model_base_path']), f\"TorchSparse model base path {base_paths['torchsparse_model_base_path']} does not exist.\"\n",
    "    assert os.path.isdir(base_paths['openpc_cfg_base_path']), f\"OpenPCDet cfg base path {base_paths['openpc_cfg_base_path']} does not exist.\"\n",
    "    assert os.path.isdir(base_paths['torchsparse_cfg_base_path']), f\"TorchSparse cfg base path {base_paths['torchsparse_cfg_base_path']} does not exist.\"\n",
    "    assert os.path.isdir(base_paths['openpc_path']), f\"OpenPCDet path {base_paths['openpc_path']} does not exist.\"\n",
    "\n",
    "    # pre-process paths\n",
    "    openpc_cfg_path = os.path.join(base_paths['openpc_cfg_base_path'], registry_entry['openpc_cfg_path'])\n",
    "    torchsparse_cfg_path = os.path.join(base_paths['torchsparse_cfg_base_path'], registry_entry['torchsparse_cfg_path'])\n",
    "    test_file_path = os.path.join(base_paths['openpc_path'], \"tools/test.py\")\n",
    "    openpc_model_path = os.path.join(base_paths['openpc_model_base_path'], registry_entry['ckpt_before'])\n",
    "    assert os.path.isdir(base_paths['torchsparse_model_base_path']), \"Please create the directory for the converted model.\"\n",
    "    torchsparse_model_path = os.path.join(base_paths['torchsparse_model_base_path'], registry_entry['ckpt_after'])\n",
    "\n",
    "    # convert the model\n",
    "    if convert:\n",
    "        parent_dir = os.path.dirname(torchsparse_model_path)\n",
    "        if not os.path.exists(parent_dir):\n",
    "            os.makedirs(parent_dir)\n",
    "        converter(\n",
    "            ckpt_before=openpc_model_path,\n",
    "            ckpt_after=torchsparse_model_path,\n",
    "            cfg_path=openpc_cfg_path,\n",
    "            v_spconv = registry_entry['v_spconv'],\n",
    "            framework='openpc'\n",
    "        )\n",
    "\n",
    "    command = f'bash -c \"conda activate {env_name}; python {test_file_path} --cfg_file {torchsparse_cfg_path} --ckpt {torchsparse_model_path}\"'\n",
    "    print(command)\n",
    "    result = subprocess.run(command, capture_output=True, text=True, shell=True, executable='/bin/bash')\n",
    "    return result  # result have .stdout and .stderr attributes to get the output. \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "### SECOND\n",
    "Run a SECOND demo. You can print the evaluation results of the model from the sub-process's `stdout` and `stderr`. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "second_registry = {\n",
    "    'ckpt_before': 'SECOND/second_7862.pth',\n",
    "    'ckpt_after': 'SECOND/second_7862.pth',\n",
    "    'openpc_cfg_path': 'kitti_models/second.yaml',\n",
    "    'torchsparse_cfg_path': 'kitti_models/second_plugin.yaml',\n",
    "    'v_spconv': 1\n",
    "}\n",
    "\n",
    "second_results = openpc_single_demo(second_registry, base_paths, convert=True)\n",
    "print(second_results.stdout)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printout the evaluation resutls. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(second_results.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected Outputs: \n",
    "\n",
    "```\n",
    "2024-08-03 01:31:45,287   INFO  *************** Performance of EPOCH 7862 *****************\n",
    "2024-08-03 01:31:45,288   INFO  Generate label finished(sec_per_example: 0.0071 second).\n",
    "2024-08-03 01:31:45,288   INFO  recall_roi_0.3: 0.000000\n",
    "2024-08-03 01:31:45,288   INFO  recall_rcnn_0.3: 0.949140\n",
    "2024-08-03 01:31:45,288   INFO  recall_roi_0.5: 0.000000\n",
    "2024-08-03 01:31:45,288   INFO  recall_rcnn_0.5: 0.890933\n",
    "2024-08-03 01:31:45,288   INFO  recall_roi_0.7: 0.000000\n",
    "2024-08-03 01:31:45,288   INFO  recall_rcnn_0.7: 0.665509\n",
    "2024-08-03 01:31:45,289   INFO  Average predicted number of objects(3769 samples): 14.170\n",
    "\n",
    "2024-08-03 01:32:02,104   INFO  Car AP@0.70, 0.70, 0.70:\n",
    "bbox AP:90.7803, 89.8974, 89.0423\n",
    "bev  AP:90.0077, 87.9303, 86.4681\n",
    "3d   AP:88.6137, 78.6221, 77.1884\n",
    "aos  AP:90.76, 89.77, 88.82\n",
    "Car AP_R40@0.70, 0.70, 0.70:\n",
    "bbox AP:95.6299, 94.1655, 91.7658\n",
    "bev  AP:92.4182, 88.5667, 87.6569\n",
    "3d   AP:90.5570, 81.6217, 78.5918\n",
    "aos  AP:95.60, 94.00, 91.52\n",
    "Car AP@0.70, 0.50, 0.50:\n",
    "bbox AP:90.7803, 89.8974, 89.0423\n",
    "bev  AP:90.7940, 90.1455, 89.5185\n",
    "3d   AP:90.7940, 90.0875, 89.3992\n",
    "aos  AP:90.76, 89.77, 88.82\n",
    "Car AP_R40@0.70, 0.50, 0.50:\n",
    "bbox AP:95.6299, 94.1655, 91.7658\n",
    "bev  AP:95.6780, 94.8539, 94.2489\n",
    "3d   AP:95.6652, 94.7514, 94.0543\n",
    "aos  AP:95.60, 94.00, 91.52\n",
    "Pedestrian AP@0.50, 0.50, 0.50:\n",
    "bbox AP:68.8175, 66.3438, 63.3219\n",
    "bev  AP:62.0900, 56.6800, 53.8592\n",
    "3d   AP:56.5254, 52.9844, 47.6520\n",
    "aos  AP:64.72, 61.77, 58.52\n",
    "Pedestrian AP_R40@0.50, 0.50, 0.50:\n",
    "bbox AP:69.5828, 66.5427, 62.8821\n",
    "bev  AP:60.8272, 56.6307, 52.1901\n",
    "3d   AP:55.8311, 51.1306, 46.0872\n",
    "aos  AP:64.94, 61.34, 57.60\n",
    "Pedestrian AP@0.50, 0.25, 0.25:\n",
    "bbox AP:68.8175, 66.3438, 63.3219\n",
    "bev  AP:75.4727, 73.9056, 69.7768\n",
    "3d   AP:75.4608, 73.8354, 69.6737\n",
    "aos  AP:64.72, 61.77, 58.52\n",
    "Pedestrian AP_R40@0.50, 0.25, 0.25:\n",
    "bbox AP:69.5828, 66.5427, 62.8821\n",
    "bev  AP:76.3837, 74.7549, 70.8290\n",
    "3d   AP:76.3760, 74.6214, 70.7453\n",
    "aos  AP:64.94, 61.34, 57.60\n",
    "Cyclist AP@0.50, 0.50, 0.50:\n",
    "bbox AP:87.5600, 77.0733, 74.3888\n",
    "bev  AP:84.0752, 70.7383, 65.4639\n",
    "3d   AP:80.6792, 67.1007, 62.1278\n",
    "aos  AP:87.42, 76.73, 74.00\n",
    "Cyclist AP_R40@0.50, 0.50, 0.50:\n",
    "bbox AP:91.4045, 79.0474, 75.4231\n",
    "bev  AP:88.0865, 71.1796, 66.8857\n",
    "3d   AP:83.0254, 66.7160, 62.5131\n",
    "aos  AP:91.24, 78.63, 74.98\n",
    "Cyclist AP@0.50, 0.25, 0.25:\n",
    "bbox AP:87.5600, 77.0733, 74.3888\n",
    "bev  AP:86.0192, 76.8562, 72.4265\n",
    "3d   AP:86.0192, 76.8562, 72.4245\n",
    "aos  AP:87.42, 76.73, 74.00\n",
    "Cyclist AP_R40@0.50, 0.25, 0.25:\n",
    "bbox AP:91.4045, 79.0474, 75.4231\n",
    "bev  AP:90.2871, 77.3294, 73.5617\n",
    "3d   AP:90.2871, 77.3293, 73.5547\n",
    "aos  AP:91.24, 78.63, 74.98\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PV-RCNN\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv_rcnn_registry = {\n",
    "    'ckpt_before': 'PV-RCNN/pv_rcnn_8369.pth',\n",
    "    'ckpt_after': 'PV-RCNN/pv_rcnn_8369.pth',\n",
    "    'openpc_cfg_path': 'kitti_models/pv_rcnn.yaml',\n",
    "    'torchsparse_cfg_path': 'kitti_models/pv_rcnn_plugin.yaml',\n",
    "    'v_spconv': 1\n",
    "}\n",
    "\n",
    "pv_rcnn_results = openpc_single_demo(pv_rcnn_registry, base_paths, convert=True)\n",
    "print(pv_rcnn_results.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printout the evaluation results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pv_rcnn_results.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected Output: \n",
    "\n",
    "```\n",
    "2024-08-03 01:36:13,184   INFO  *************** Performance of EPOCH 8369 *****************\n",
    "2024-08-03 01:36:13,184   INFO  Generate label finished(sec_per_example: 0.0375 second).\n",
    "2024-08-03 01:36:13,184   INFO  recall_roi_0.3: 0.968447\n",
    "2024-08-03 01:36:13,184   INFO  recall_rcnn_0.3: 0.968504\n",
    "2024-08-03 01:36:13,184   INFO  recall_roi_0.5: 0.928352\n",
    "2024-08-03 01:36:13,184   INFO  recall_rcnn_0.5: 0.934389\n",
    "2024-08-03 01:36:13,184   INFO  recall_roi_0.7: 0.716938\n",
    "2024-08-03 01:36:13,184   INFO  recall_rcnn_0.7: 0.759369\n",
    "2024-08-03 01:36:13,186   INFO  Average predicted number of objects(3769 samples): 9.206\n",
    "\n",
    "2024-08-03 01:36:31,071   INFO  Car AP@0.70, 0.70, 0.70:\n",
    "bbox AP:96.2503, 89.5026, 89.2371\n",
    "bev  AP:90.0794, 87.9057, 87.4102\n",
    "3d   AP:89.3399, 83.7288, 78.7346\n",
    "aos  AP:96.23, 89.39, 89.07\n",
    "Car AP_R40@0.70, 0.70, 0.70:\n",
    "bbox AP:98.2628, 94.4161, 92.2756\n",
    "bev  AP:93.0160, 90.3269, 88.5353\n",
    "3d   AP:92.1238, 84.4006, 82.4978\n",
    "aos  AP:98.24, 94.26, 92.07\n",
    "Car AP@0.70, 0.50, 0.50:\n",
    "bbox AP:96.2503, 89.5026, 89.2371\n",
    "bev  AP:96.2815, 89.4945, 89.2827\n",
    "3d   AP:96.2358, 89.4761, 89.2522\n",
    "aos  AP:96.23, 89.39, 89.07\n",
    "Car AP_R40@0.70, 0.50, 0.50:\n",
    "bbox AP:98.2628, 94.4161, 92.2756\n",
    "bev  AP:98.2553, 94.5899, 94.4293\n",
    "3d   AP:98.2372, 94.5246, 94.3279\n",
    "aos  AP:98.24, 94.26, 92.07\n",
    "Pedestrian AP@0.50, 0.50, 0.50:\n",
    "bbox AP:73.1750, 68.1068, 64.3705\n",
    "bev  AP:65.1549, 59.4407, 54.5509\n",
    "3d   AP:63.2132, 54.8977, 51.9049\n",
    "aos  AP:67.88, 62.55, 58.76\n",
    "Pedestrian AP_R40@0.50, 0.50, 0.50:\n",
    "bbox AP:73.6830, 68.3395, 64.3856\n",
    "bev  AP:65.9740, 58.5465, 54.1692\n",
    "3d   AP:62.8141, 54.5642, 49.9484\n",
    "aos  AP:67.88, 62.23, 58.11\n",
    "Pedestrian AP@0.50, 0.25, 0.25:\n",
    "bbox AP:73.1750, 68.1068, 64.3705\n",
    "bev  AP:76.3273, 71.9281, 69.4853\n",
    "3d   AP:76.3142, 71.8653, 69.4437\n",
    "aos  AP:67.88, 62.55, 58.76\n",
    "Pedestrian AP_R40@0.50, 0.25, 0.25:\n",
    "bbox AP:73.6830, 68.3395, 64.3856\n",
    "bev  AP:78.3134, 73.1922, 70.0144\n",
    "3d   AP:78.2993, 73.0964, 69.9333\n",
    "aos  AP:67.88, 62.23, 58.11\n",
    "Cyclist AP@0.50, 0.50, 0.50:\n",
    "bbox AP:96.2518, 81.3697, 76.5945\n",
    "bev  AP:88.5292, 73.4151, 70.4412\n",
    "3d   AP:86.2533, 69.8507, 64.5986\n",
    "aos  AP:96.11, 81.08, 76.25\n",
    "Cyclist AP_R40@0.50, 0.50, 0.50:\n",
    "bbox AP:97.1791, 82.4359, 78.2700\n",
    "bev  AP:93.4740, 74.6551, 70.2368\n",
    "3d   AP:90.8024, 70.5917, 66.0683\n",
    "aos  AP:97.07, 82.13, 77.92\n",
    "Cyclist AP@0.50, 0.25, 0.25:\n",
    "bbox AP:96.2518, 81.3697, 76.5945\n",
    "bev  AP:95.1249, 78.3304, 73.3570\n",
    "3d   AP:95.1249, 78.3214, 73.2931\n",
    "aos  AP:96.11, 81.08, 76.25\n",
    "Cyclist AP_R40@0.50, 0.25, 0.25:\n",
    "bbox AP:97.1791, 82.4359, 78.2700\n",
    "bev  AP:96.2788, 79.1643, 75.9079\n",
    "3d   AP:96.2788, 79.1586, 75.8420\n",
    "aos  AP:97.07, 82.13, 77.92\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part-A2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "part_a2_registry = {\n",
    "    'ckpt_before': 'Part-A2/PartA2_7940.pth',\n",
    "    'ckpt_after': 'Part-A2/PartA2_7940.pth',\n",
    "    'openpc_cfg_path': 'kitti_models/PartA2.yaml',\n",
    "    'torchsparse_cfg_path': 'kitti_models/PartA2_plugin.yaml',\n",
    "    'v_spconv': 1\n",
    "}\n",
    "\n",
    "part_a2_results = openpc_single_demo(part_a2_registry, base_paths, convert=True)\n",
    "print(part_a2_results.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected Output: \n",
    "\n",
    "```\n",
    "2024-08-04 22:25:58,225   INFO  *************** Performance of EPOCH 7940 *****************\n",
    "2024-08-04 22:25:58,225   INFO  Generate label finished(sec_per_example: 0.0148 second).\n",
    "2024-08-04 22:25:58,225   INFO  recall_roi_0.3: 0.970270\n",
    "2024-08-04 22:25:58,225   INFO  recall_rcnn_0.3: 0.970384\n",
    "2024-08-04 22:25:58,225   INFO  recall_roi_0.5: 0.930117\n",
    "2024-08-04 22:25:58,225   INFO  recall_rcnn_0.5: 0.935870\n",
    "2024-08-04 22:25:58,225   INFO  recall_roi_0.7: 0.710901\n",
    "2024-08-04 22:25:58,225   INFO  recall_rcnn_0.7: 0.746953\n",
    "2024-08-04 22:25:58,227   INFO  Average predicted number of objects(3769 samples): 11.203\n",
    "\n",
    "2024-08-04 22:26:14,556   INFO  Car AP@0.70, 0.70, 0.70:\n",
    "bbox AP:94.7980, 89.3054, 89.0451\n",
    "bev  AP:90.2376, 87.9529, 87.5332\n",
    "3d   AP:89.5301, 79.3967, 78.8623\n",
    "aos  AP:94.74, 89.15, 88.82\n",
    "Car AP_R40@0.70, 0.70, 0.70:\n",
    "bbox AP:97.8505, 93.7762, 91.7622\n",
    "bev  AP:92.8941, 89.9358, 88.3568\n",
    "3d   AP:92.1135, 82.8972, 82.1563\n",
    "aos  AP:97.79, 93.58, 91.50\n",
    "Car AP@0.70, 0.50, 0.50:\n",
    "bbox AP:94.7980, 89.3054, 89.0451\n",
    "bev  AP:94.7820, 89.3003, 89.0972\n",
    "3d   AP:94.7357, 89.2693, 89.0323\n",
    "aos  AP:94.74, 89.15, 88.82\n",
    "Car AP_R40@0.70, 0.50, 0.50:\n",
    "bbox AP:97.8505, 93.7762, 91.7622\n",
    "bev  AP:97.7269, 93.9681, 93.9136\n",
    "3d   AP:97.7006, 93.8868, 93.7499\n",
    "aos  AP:97.79, 93.58, 91.50\n",
    "Pedestrian AP@0.50, 0.50, 0.50:\n",
    "bbox AP:75.7000, 71.2206, 67.4326\n",
    "bev  AP:71.0432, 64.1691, 60.1552\n",
    "3d   AP:65.8572, 60.2919, 55.6764\n",
    "aos  AP:73.64, 68.58, 64.55\n",
    "Pedestrian AP_R40@0.50, 0.50, 0.50:\n",
    "bbox AP:76.2101, 71.9826, 68.6716\n",
    "bev  AP:70.8884, 64.3507, 59.3637\n",
    "3d   AP:67.2230, 59.9240, 54.7550\n",
    "aos  AP:73.94, 69.13, 65.41\n",
    "Pedestrian AP@0.50, 0.25, 0.25:\n",
    "bbox AP:75.7000, 71.2206, 67.4326\n",
    "bev  AP:78.8260, 75.3293, 72.8212\n",
    "3d   AP:78.8474, 75.2618, 72.7057\n",
    "aos  AP:73.64, 68.58, 64.55\n",
    "Pedestrian AP_R40@0.50, 0.25, 0.25:\n",
    "bbox AP:76.2101, 71.9826, 68.6716\n",
    "bev  AP:81.3466, 77.4266, 73.8207\n",
    "3d   AP:81.4174, 77.2437, 73.6680\n",
    "aos  AP:73.94, 69.13, 65.41\n",
    "Cyclist AP@0.50, 0.50, 0.50:\n",
    "bbox AP:89.0268, 77.5214, 76.1501\n",
    "bev  AP:87.2364, 73.7817, 70.8660\n",
    "3d   AP:85.6864, 70.1335, 65.5218\n",
    "aos  AP:88.87, 77.18, 75.69\n",
    "Cyclist AP_R40@0.50, 0.50, 0.50:\n",
    "bbox AP:94.3109, 80.2203, 77.5290\n",
    "bev  AP:92.3125, 73.8489, 70.7336\n",
    "3d   AP:90.5849, 70.2803, 67.1889\n",
    "aos  AP:94.10, 79.78, 77.05\n",
    "Cyclist AP@0.50, 0.25, 0.25:\n",
    "bbox AP:89.0268, 77.5214, 76.1501\n",
    "bev  AP:87.7328, 77.7907, 73.4163\n",
    "3d   AP:87.7328, 77.7888, 73.3536\n",
    "aos  AP:88.87, 77.18, 75.69\n",
    "Cyclist AP_R40@0.50, 0.25, 0.25:\n",
    "bbox AP:94.3109, 80.2203, 77.5290\n",
    "bev  AP:92.7067, 78.4131, 75.5830\n",
    "3d   AP:92.7029, 78.4100, 75.4983\n",
    "aos  AP:94.10, 79.78, 77.05\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VoxelNeXt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weights of VoxelNeXt differs from the previous models because it contains 2D sparse conv layers. We have a specialized converter for it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from converter_voxelnext import convert_voxelnext\n",
    "\n",
    "voxelnext_registry = {\n",
    "    'ckpt_before': 'VoxelNeXt/voxelnext_nuscenes_kernel1.pth',\n",
    "    'ckpt_after': 'VoxelNeXt/voxelnext_nuscenes_kernel1.pth',\n",
    "    'openpc_cfg_path': 'nuscenes_models/cbgs_voxel0075_voxelnext.yaml',\n",
    "    'torchsparse_cfg_path': 'nuscenes_models/cbgs_voxel0075_voxelnext.yaml',\n",
    "    'v_spconv': 2\n",
    "}\n",
    "\n",
    "# pre-process paths\n",
    "openpc_cfg_path = os.path.join(base_paths['openpc_cfg_base_path'], voxelnext_registry['openpc_cfg_path'])\n",
    "torchsparse_cfg_path = os.path.join(base_paths['torchsparse_cfg_base_path'], voxelnext_registry['torchsparse_cfg_path'])\n",
    "test_file_path = os.path.join(base_paths['openpc_path'], \"tools/test.py\")\n",
    "openpc_model_path = os.path.join(base_paths['openpc_model_base_path'], voxelnext_registry['ckpt_before'])\n",
    "torchsparse_model_path = os.path.join(base_paths['torchsparse_model_base_path'], voxelnext_registry['ckpt_after'])\n",
    "\n",
    "# convert_voxelnext(\n",
    "#     ckpt_before=openpc_model_path,\n",
    "#     ckpt_after=torchsparse_model_path,\n",
    "# )\n",
    "\n",
    "command = f'bash -c \"conda activate torchsparse; python {test_file_path} --cfg_file {torchsparse_cfg_path} --ckpt {torchsparse_model_path}\"'\n",
    "voxelnext_results = subprocess.run(command, capture_output=True, text=True, shell=True, executable='/bin/bash')\n",
    "\n",
    "print(voxelnext_results.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected Output: \n",
    "\n",
    "```\n",
    "2024-08-05 00:49:37,228   INFO  ----------------Nuscene detection_cvpr_2019 results-----------------\n",
    "***car error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.19, 0.16, 0.17, 0.23, 0.20 | 73.06, 84.46, 88.36, 89.65 | mean AP: 0.8388369993093885\n",
    "***truck error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.33, 0.18, 0.17, 0.20, 0.22 | 36.22, 55.39, 63.48, 67.13 | mean AP: 0.555570448511005\n",
    "***construction_vehicle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.70, 0.43, 1.22, 0.12, 0.30 | 2.29, 12.86, 27.62, 41.68 | mean AP: 0.21110277861079582\n",
    "***bus error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.34, 0.18, 0.10, 0.38, 0.24 | 44.01, 71.78, 81.83, 84.45 | mean AP: 0.7051496836594204\n",
    "***trailer error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.60, 0.20, 0.63, 0.16, 0.18 | 7.27, 32.79, 51.76, 60.86 | mean AP: 0.381712143855136\n",
    "***barrier error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.20, 0.27, 0.09, nan, nan | 59.00, 69.85, 73.59, 75.16 | mean AP: 0.6940105701286209\n",
    "***motorcycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.20, 0.24, 0.32, 0.31, 0.24 | 55.78, 64.17, 65.33, 65.97 | mean AP: 0.6281188558655376\n",
    "***bicycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.17, 0.27, 0.48, 0.14, 0.01 | 47.51, 50.35, 50.84, 51.43 | mean AP: 0.5003450990308462\n",
    "***pedestrian error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.15, 0.27, 0.47, 0.20, 0.09 | 82.28, 83.92, 85.31, 86.79 | mean AP: 0.8457525135254048\n",
    "***traffic_cone error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0\n",
    "0.14, 0.32, nan, nan, nan | 66.33, 67.87, 69.92, 73.18 | mean AP: 0.6932564816100588\n",
    "--------------average performance-------------\n",
    "trans_err:\t 0.3013\n",
    "scale_err:\t 0.2522\n",
    "orient_err:\t 0.4058\n",
    "vel_err:\t 0.2169\n",
    "attr_err:\t 0.1856\n",
    "mAP:\t 0.6054\n",
    "NDS:\t 0.6665\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torchsparse",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
